Real Time Loading of Enterprise Data Using Fragmentation of Data Warehouses

Real-time ETL (Extraction, Transformation and Loading) of enterprise data is one of the foremost features of next generation Business Intelligence (BI 2.0). This paper presents a proposal for loading operational data in real time using a Data Warehouse (DW) architecture with faster processing time than current approaches. Distributed database techniques, like derived horizontal fragmentation in shared nothing architecture, are used to create fragments that are specialized in most current data and optimized to achieve continuous insertions. Using this approach, the DW can be updated near-line from operational data sources. As a result, DW queries are executed over real time data or very close to that. Moreover, continuous loadings do not impact queries response time. In addition, we extended the Star Schema Benchmark to address loading operational data in real time. This benchmark, including the new feature, is used to validate and demonstrate the efficiency of our approach compared to other ones.